Field of the Invention
The present invention relates to an image pickup apparatus and an image processing apparatus which are capable of generating a restored image from a captured image.
Description of the Related Art
A captured image obtained by an image pickup apparatus is deteriorated due to a diffraction phenomenon depending on an aperture value (F number) even when each aberration such as a spherical aberration, a coma aberration, field curvature, and astigmatism is corrected with high accuracy.
FIG. 5 is a diffraction limit curve, and the horizontal axis and the vertical axis indicate a spatial frequency and a MTF, respectively. As illustrated in FIG. 5, the darker color the F number indicates, to the lower frequency side the cutoff frequency is shifted. For instance, the Nyquist frequency of an image sensor with a pixel size of 4 μm is 125 lines/mm. Accordingly, when an F number, for example F2.8, indicates a bright color, the degree of adverse effects caused by such a diffraction phenomenon is small. By contrast, when an F number, for example F16 or F32, indicates a dark color, the degree of such adverse effects is large. Since the diffraction phenomenon can be represented by an OTF or a PSF as in the case of the aberration, image restoration processing can correct a blur due to the diffraction.
The presence of a blur component of an image caused by the aberration or diffraction means that when a light beam emitted from one point does not suffer from any aberration or diffraction, the light beam fails to converge to another point, being minutely spread around the another point. Such a minutely-spread distribution is represented by a point spread function (PSF).
An optical transfer function (OTF) obtained by the Fourier transform of the PSF is frequency component information on an aberration and represented by a complex number. An absolute value of the OTF, i.e., an amplitude component, is referred to as a “modulation transfer function” (MTF), and a phase component is referred to as a “phase transfer function” (PTF). The modulation component MTF and the phase component PTF are an amplitude component of image deterioration caused by an aberration, and frequency characteristics of a phase component, respectively, and represented by an expression below with the phase component being defined as a phase angle.PTF=tan−1(Im(OTF)/Re(OTF))
In this expression, symbols Re (OTF) and Im (OTF) represent the real part and the imaginary part of an OTF, respectively. As a method of correcting a deterioration of the amplitude component MTF and the phase component PTF, the correction with the use of information on an OTF of an image pickup optical system is known. This method is typically called image restoration or image reconstruction. Accordingly, processing in which a deteriorated captured image is corrected by using information on the OTF of the image pickup optical system is hereinafter referred to as “image restoration processing”. As one of the image restoration methods, the method of convolving an image restoration filter with reverse characteristics of an optical transfer function (OTF) with respect to an input image, which is described below, is known.
The effective use of a restored image requires obtaining more accurate OTF information of the image pickup optical system. For example, the OTF can be obtained by calculation based on design value information of the image pickup optical system. The OTF can be also obtained by capturing a point light source and then performing the Fourier transform on its intensity distribution. Furthermore, for the diffraction, it can be obtained by a theoretically-derived calculation expression.
Japanese Patent Laid-open No. 2012-73691 discloses a method of performing the image restoration processing according to various image capturing conditions of an image pickup apparatus while storing an OTF for use in image restoration which is converted into a coefficient. However, the OTF varies depending on an aperture value, a photographing distance, and a focal length of a zoom lens. Furthermore, the OTF varies depending on a position in an image (image height). Accordingly, entire highly-accurate OTF data has a large amount of data. When the image restoration processing is performed for a single image, the OTF is used while changing depending on the image height and accordingly an amount of calculation is large. The method disclosed in Japanese Patent Laid-open No. 2012-73691 reduces an amount of data by storing an OTF for use in image restoration which is converted into a coefficient. However, further reduction in amounts of data and calculation is required since they affect a calculation speed of an image processing circuit of the image pickup apparatus and a cost for a memory of the image pickup apparatus.
Japanese Patent Laid-open No. 2014-150423 discloses an image pickup apparatus which selects an appropriate image restoration filter depending on an aperture value (F number) when the aperture value is large, and thus it is capable of performing image restoration processing with high accuracy while reducing in amounts of data and calculation.
International Publication No. WO2011/118077 discloses a method of applying an image restoration filter to each of a plurality of parallax images to perform optical correction processing (image restoration processing).
However, in the method disclosed in International Publication No. WO2011/118077, the optical correction processing is performed before the plurality of parallax images are synthesized (i.e., refocus processing is performed), and accordingly it is necessary to perform the optical correction processing on each of the plurality of parallax images. Consequently, the amount of data of the image restoration filter and the amount of calculation of the optical correction processing (image restoration processing) increase.